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Abstract: This paper considers the fixed point problem for a nonexpansive mapping on 
a real Hilbert space and proposes novel line search fixed point algorithms to accelerate the 
search. The termination conditions for the line search are based on the well-known Wolfe 
conditions that are used to ensure the convergence and stability of unconstrained optimiza¬ 
tion algorithms. The directions to search for fixed points are generated by using the ideas of 
the steepest descent direction and conventional nonlinear conjugate gradient directions for 
unconstrained optimization. We perform convergence as well as convergence rate analyses 
on the algorithms for solving the fixed point problem under certain assumptions. The main 
contribution of this paper is to make a concrete response to an issue of constrained smooth 
convex optimization; that is, whether or not we can devise nonlinear conjugate gradient 
algorithms to solve constrained smooth convex optimization problems. We show that the 
proposed fixed point algorithms include ones with nonlinear conjugate gradient directions 
which can solve constrained smooth convex optimization problems. To illustrate the practi¬ 
cality of the algorithms, we apply them to concrete constrained smooth convex optimization 
problems, such as constrained quadratic programming problems and generalized convex fea¬ 
sibility problems, and numerically compare them with previous algorithms based on the 
Krasnosel’skii-Mann fixed point algorithm. The results show that the proposed algorithms 
dramatically reduce the running time and iterations needed to find optimal solutions to the 
concrete optimization problems compared with the previous algorithms. 
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1 Introduction 

Consider the following fixed point problem (see O Chapter 4], [121 Chapter 3], [131 Chapter 
1], [33l Chapter 3]): 

Find X* G Fix (T) ■= {x* € H ■. T {x*) = x*} , (1.1) 

where H stands for a real Hilbert space with inner product (•, •) and its induced norm || • ||, T 
is a ?7,onea;pansjve mapping from iJ into itself (i.e., ||T(a;) — T(?/)|| < ||a: —y|| (a:, j/G i7)), and 
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one assumes Fix(r) ^ 0. Problem (11.11) includes convex feasibility problems [2], [3l Example 
5.21], constrained smooth convex optimization problems [33 Proposition 4.2], problems of 
finding the zeros of monotone operators [3] Proposition 23.38], and monotone variational 
inequalities [3j Subchapter 25.5]. 

There are useful algorithms for solving Problem (HU, such as the Krasnosel’skii-Mann 
algorithm m Subchapter 5.2], |4l Subchapter 1.2], [8l [2^ [25]. Halpern algorithm jH Sub¬ 
chapter 1.2], [T3|34], and hybrid method [26] (Solodov and Svaiter [32] proposed the hybrid 
method to solve problems of finding the zeros of monotone operators). This paper focuses 
on the Krasnosel’skh-Mann algorithm that has practical applications, such as analyses of 
dynamic systems governed by maximal monotone operators [3 and nonsmooth convex vari¬ 
ational signal recovery [3, defined as follows: given the current iterate Xn & H and step size 
an € [0,1], the next iterate Xn+i of the algorithm is 

Xn+l := Xn + an {T (Xn) - Xn) ■ (1-2) 


Assuming that (Q!n)neN satisfies the condition. 


OO 

^ ^ 0(7T,(l O^n) — CK), 
n—0 


(1.3) 


the sequence (xn)neN generated by Algorithm (HU weakly converges to a fixed point of T 
(see, e.g., [S] Theorem 5.14]). This result indicates that Algorithm (11.21) with constant step 
sizes (e.g., an ■= a € (0,1) {n G N)) or diminishing step sizes (e.g., := l/(n + l) (n G N)) 

can solve Problem (HU. Propositions 10 and 11 in [8] indicate that Algorithm (|1.2I) with 
Condition (HU has the following rate of convergence: for all n G N, 


\\xn - T (a;„)l| = O 



Ofc (1 - Ofc) 


(1.4) 


(e.g., l|a:„ — r(a:„)l| = Oiljyjn + 1) when := a G (0,1) (n G N)). This fact implies that 
Algorithm (11.21) with (HU does not always have fast convergence and has motivated the 
development of modifications and variants for the Krasnosel’skii-Mann algorithm in order 
to accelerate Algorithm (HU- 

One approach to accelerate Algorithm (11.21) with (11.31) is to develop line search methods 
that can determine a more adequate step size than a step size satisfying (HU at each 
iteration n so that the value of l|a:„+i — r(a:„+i)jj decreases dramatically. Magnanti and 
Perakis proposed an adaptive line search framework [241 Section 2] that can determine step 
sizes to satisfy weaker conditions [24| Assumptions A1 and A2] than ()1.3I) . On the basis 
of this framework, they showed that Algorithm (II. 2L with step sizes an satisfying the 
following Armijo-type condition, converges to a fixed point of T [241 Theorems 4 and 8]: 
given Xn G /3 > 0,D > 0, and b G (0,1), choose the smallest nonnegative integer In so 
that a„ = 6*" satisfies the condition, 

g-a {an) - gn (0) < - 06 *" JlT {Xn) - Xn\f , (1-5) 

where gn '■ [0,1] —>■ M is a potential function [24] Scheme IV] defined for all a G [0,1] by 

gn (a) := Jj (Xn -h a (T (xn) - x„)) - T (xn -h a (T (xn) - Xn)) IP 
- Pa{l- a) \\T (xn) - x„|p. 
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Theorem 5 in [24] shows that Algorithm (EU with the Armijo-type condition (11.51) satisfies 
that ll^n-i-i — T{xn+i)\\'^ ^ [1 ~ Pictn — l/2)^]||a;n — T{xn)\\^ {n G N), which implies that the 
algorithm has that, for all n G N, 



In this paper, we introduce a line search framework using defined by dm), dm, and 
(ll.lOp . which is the simplest of all potential functions including gn defined as in dm: given 
dn G H, for all a G [0,1], 


Xn{a) := Xn + adn, ( 1 . 8 ) 

Qn{a) := Xn{a)-T{xn{a)), (1.9) 

Pn{a):=\\Qn{a)f. (1.10) 

When dn ■= —(xn — T(xn)) and is given as in (|1.3I1 . the point Xn(an) in coin¬ 

cides with Xn+i defined by Algorithm (11.21) with (11.31) . Consider the following problem of 
minimizing over [0,1]: 

Find a„ G [0,1] such that (a„) = min P„ (a). (l-H) 

When the solution a„ to Problem (ll.ll|) can be obtained in each iteration, P„(an) < Cra(O) 
holds for all n G N. Accordingly, if the next iterate Xn+i is defined by Xn+i '■= Xni^n), 
\\xn+i - T{x„+i)\\ < \\xn - T{xn)\\ (u G N) ho lds, L e., (||a;„ - r(x„)||) is monotone 
decreasing. Since the exact solution to Problem (11.111) cannot be easily obtained, the step 
size an can be chosen so as to yield an approximate minimum for Problem (11.111) in each 
iteration, specifically, to satisfy the following Wolfe-type conditions [35l|36]: given Xn,dn G 
H, and 6,a G (0,1) with 6 < a, 


Pn {an) - Pn (0) < (5a„ {Qn (0), d„), (1.12) 

{Qn{an) ,dn) >(j {Qn{Q) ,dn) ■ (1.13) 


Condition ()1.12|) is the Armijo-type condition for P„ (see (|1.5I) for the Armijo-type 
condition with dn '■= —{xn — T{xn)) for the potential function g„). Under the conditions 
that dn '■= —{Xn — T{xn)) and Xn+i '■= Xn{an) (n G N), Algorithm (II.2p with (11.121) satisfies 
||a;„+i — T(a;„+i)|p < (1 — (5a„)||a;„ — T(x„)|p (n G N), which implies that, for all n G N0 


ll^n 


T{Xn)\\=0 




(1.14) 


Here, let us see how the step size conditions (11.31) . (11.51) . (11.121) . and (11.131) affect the 
efficiency of Algorithm (11.21) . Algorithm (11.21) with (11.31) satisfies |la;„+i — T{xn+i)\\‘^ < 
\\xn — T{xn)\\‘^ (n G N) m (5.14)], while Algorithm (11.21) with each of (11.51) and (11.121) 
satisfies ||a:„+i — T(x„+i)|p < \\xn — T{xn)\\^ {n G N). Hence, it can be expected that 
Algorithm (11.21) with each of (ll.5|) and (|1.12l) performs better than Algorithm (11.21) with 

^See Theorem I2.6l il for the details of the convergence rate of the proposed algorithm 
when dn '■= —(x„ — T{xn)) {n G N). 
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m- Since the Armijo-type conditions (USD and (11.121) are satisfied for all sufficiently 
small values of an m Subchapter 3.1], there is a possibility that Algorithm (|1.2I) with 
only the Armijo-type condition (II.5p does not make reasonable progress. Meanwhile, (11.131) 
based on the curvature condition m Subchapter 3.1] is used to ensure that an is not too 
small and that unacceptably short steps are ruled out. Therefore, the Wolfe-type conditions 
(I1.12P and (11.131) should be used to secure efficiency of the algorithm. Moreover, even when 
an satisfying m is not small enough, it can be expected that Algorithm (Eli) with the 
Wolfe-type conditions (11.121) and (11.131) will have a better convergence rate than Algorithm 
EU with the Armijo-type condition (11.51) because of (11.71) . (11.141) . and (a — 1/2)^ < a 
{a G [(2 — •\/3)/2,1]). Section [3] introduces the line search algorithm Algorithm 4.6] to 
compute step sizes satisfying (11.121) and (11.131) with appropriately chosen 6 and a and gives 
performance comparisons of Algorithm EH with each of m and E3 with the one with 
(fTT^ and (frT3l) . 

The main concern regarding this line search is how the direction should be updated 
to accelerate the search for a fixed point of T. To address this concern, the following problem 
will be discussed: 


Minimize / (x) subject to x € H, 


(1.15) 


where /: —>■ R is convex and Frechet differentiable and V/: H H is Lipschitz contin¬ 

uous with a constant L. Let us define : H ^ H hy 

:= Id - AV/, (1.16) 


where Id stands for the identity mapping on H and A > 0. The mapping satisfies 
the nonexpansivity condition for A G (0, 2/L] [TQ] Proposition 2.3] and Fix(T(^)) coincides 
with the solution set of Problem (|1.15p . From T^^\x) — x = {x — AV/(x)) — x = —AV/(x) 
(A > 0,x G H), Algorithm (|1.2I) for solving Problem (11.151) is 

Xn-\-l — Xn an (x^^) Xn^^ Xn ^Oin'^f (x^) . (1.17) 

This means that the direction dn'^ := — (x„ — T^^\xn)) = — AV/(x„) is the steepest descent 
direction of / at Xn and Algorithm (11.21) with (i.e.. Algorithm (11.171) ') is the steepest 
descent method [371 Subchapter 3.3] for Problem (ll.lSp . 

There are many algorithms with useful search directions EZl Chapters 5-19] to accelerate 
the steepest descent method for unconstrained optimizations. In particular, algorithms with 
nonlinear conjugate gradient directions US], m Subchapter 5.2], 

dnh ■■= -y f {Xn+l) + Pnd[!\ (1.18) 


where Pn G M, have been widely used as efficient accelerated versions for most gradient 
methods. Well-known formulas for /3„ include the Hestenes-Stiefel (HS) [18], Fletcher- 
Reeves (FR) [10], Polak-Ribiere-Polyak (PRP) [29l|30], and Dai-Yuan (DY) [9] formulas: 


aHS (V/(x„+i),y„) 

^FR 

Pn ■ /j \ ’ Pn 

{dn^Vn) 

dPKF {Wf{Xn+l),yn) 

” ■ l|V/(x„)f ’ " 


\\^f{a:n+i)f 

l|V/(x„)f ’ 
\\^f{Xn+l)f ^ 

{dm Un) 


where := V/(x„+i) - V/(x„). 


(1.19) 
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Motivated by these observations, we decided to use the following direction to accelerate 
the search for a fixed point of T, which can be obtained by replacing V/ in (|1.18|1 with 
Id —r (see also (11.161) for the relationship between V/ and given the current direction 

dn & H, the current iterate Xn & H, and a step size satisfying (|1.12|1 and (I1.13|l . the next 
direction dn+i is defined by 

dn+l ■— ip^n+l (^n+l)) jdjidjit 

where /3„ is given by one of the formulas in (I1.19|) when V/ = Id — T. 

This paper proposes iterative algorithms (Algorithm 12.11) that use the direction (ll.20p 
and step sizes satisfying the Wolfe-type conditions (11.121) and (11.131) for solving Problem 
(HH) and describes their convergence analyses (Theorems 12.111^751) . We also provide their 
convergence rate analyses (Theorem 12.61) . 

The main contribution of this paper is to enable us to propose nonlinear conjugate 
gradient algorithms for constrained smooth convex optimization which are examples of the 
proposed line search fixed point algorithms, in contrast to the previously reported results for 
nonlinear conjugate gradient algorithms for unconstrained smooth nonconvex optimization 
mi Subchapter 5.2], [nmniiiiiisiiiiiiiisn]- Concretely speaking, our nonlinear conjugate 
gradient algorithms are obtained in the following steps. Given a nonempty, closed convex 
set C <Z H and a convex function with the Lipschitz continuous gradient, let us 

define 

T := Pc (Id - AV/), 

where A G (0, 2/L], L is the Lipschitz constant of V/, and Pc stands for the metric projection 
onto C. Then, Proposition 2.3 in m indicates that the mapping T is nonexpansive and 
satisfies 


Fix (T) = argmin/ (x). 
xec 

From (11.201) with T := Pc(Id— AV/), the proposed nonlinear conjugate gradient algorithms 
for finding a point in Fix(T) = argmin,j,g(;j f(x) can be expressed as follows: given a;„, dn G H 
and an satisfying (11.121) and (I1.13|) . 

^n-t-l ■— (c^n) — T ^ndn^ 

dn+1 ■— (^n+1 i^n+1 AV/ (x^_|_i))) -f jdndn-, 
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where /3„ G R is each of the following formulas H 

«HS+ „ / i^n+i - Pc {Xn+I “ AV/ (a;„+i)), yn) \ 

l {dn,yn) J 

^FR _ \\Xn+l - Pc {Xn+1 - AV/ (a:n+l))||^ 

" ■ \\Xn-PciXn-XVfiXn))f ’ 

«PRP+ „ i {x„+i - AV/ (a;„+i)) , yn) „ 

1 \\Xn-Pc{Xn-XVf{Xn))f 
oDY_ ll^n+1 ~ Pc {xn +1 - AV/ (a;„+i))f 

Pn ' /J \ ’ 

(«n, J/n) 

where 2 /„ := {xn+i - Pcixn+i - fixn+i))) - [xn - Pc{xn - XVf{xn))). Our convergence 
analyses are performed by referring to useful results on unconstrained smooth nonconvex 
optimization (see [U (H [HI |T6| |35l [Ml EH] and references therein) because the proposed 
fixed point algorithms are based on the steepest descent and nonlinear conjugate gradient 
directions for unconstrained smooth nonconvex optimization (see (I1.15l) - ()1.20|) b We would 
like to emphasize that combining unconstrained smooth nonconvex optimization theory with 
fixed point theory for nonexpansive mappings enables us to develop the novel nonlinear 
conjugate gradient algorithms for constrained smooth convex optimization. The nonlinear 
conjugate gradient algorithms are a concrete response to the issue of constrained smooth 
convex optimization that is whether or not we can present nonlinear conjugate gradient 
algorithms to solve constrained smooth convex optimization problems. 

To verify whether the proposed nonlinear conjugate gradient algorithms are accelera¬ 
tions for solving practical problems, we apply them to constrained quadratic programming 
problems (Subsection (HU) and generalized convex feasibility problems fSubsection 13.31) (see 
ElEZ] and references therein for the relationship between the generalized convex feasibility 
problem and signal processing problems), which are constrained smooth convex optimization 
problems and particularly interesting applications of Problem (HID. Moreover, we numeri¬ 
cally compare their abilities to solve concrete constrained quadratic programming problems 
and generalized convex feasibility problems with those of previous algorithms based on the 
Krasnosel’skii-Mann algorithm (Algorithm (11.21) with step sizes satisfying (11.31) and Algo¬ 
rithm (HID with step sizes satisfying (11.51) 1 and show that they can find optimal solutions 
to these problems faster than the previous ones. 

Throughout this paper, we shall let N be the set of zero and all positive integers, 
be a d-dimensional Euclidean space, H he a real Hilbert space with inner product (•, •) and 
its induced norm || • ||, and T.H H he a nonexpansive mapping with Fix(T) := {x G 
H: T{x) = x}^%. 

^To guarantee the convergence of the PRP and HS methods for unconstrained optimiza¬ 
tion, the formulas := max{/3™^,0} and := max{,0))^®, 0} were presented in 

m- We use the modifications to perform the convergence analyses on the proposed line 
search fixed point algorithms. 
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2 Line search fixed point algorithms based on 
nonlinear conjugate gradient directions 


Let us begin by explicitly stating our algorithm for solving Problem CH) discussed in Section 

[II 


Algorithm 2.1. 

Step 0. Take S,a € (0,1) with S < a. Choose Xq € H arbitrarily and set dg := 
— {xq — T(xo)) and n := 0. 

Step 1. Compute S (0,1] satisfying 


\\xn (an) - T (Xn (a„))|l^ - \\xn - T (x„)||^ < (5a„ {xn - T (Xn ), d„) 
(Xn {an) - T {Xn {an)) , d„) > (T {Xn “ T {Xn) , dn) , 

where Xn{an) '■= Xn + andn- Compute Xn+i € H by 

Xn-{-l •— Xn andn- 

Step 2. If ||a;„+i — T(x„+i)|| = 0, stop. Otherwise, go to Step 3. 

Step 3. Compute /3„ S R using each of the following formulas: 


pSD 


:= 0 , 


qHS+ .J (^"+1 T {Xn+l) ^Vn) qI yjFR ll^n+1 d^(^n+l)|| 


:= max • 


{dn, yn) 

{Xn+1 - T{Xn+l),yn) 
\\Xn - T {Xn)\f 


\\Xn - T (a;„)|| 


2 ’ 


/oDY ^ ||x„+i -T{Xn+l)f 
1 ’^” ■ {dn,yn) 


( 2 . 1 ) 

( 2 . 2 ) 

(2.3) 


(2.4) 


where yn ■= {xn+i - T{xn+i)) - {Xn - T{Xn))- Generate d„+i e H by 


dn+l '■— {Xn+1 T (x 72 -|_i)) + fdndn- 


Step 4- Put n := n + 1 and go to Step 1. 

We need to use appropriate line search algorithms to compute q;„ {n € N) satisfying 
m and (12.2|) . In Section jS] we use a useful one fAlgorithm 13.11) Algorithm 4.6] 
that can obtain the step sizes satisfying (EB and (12.21) whenever the line search algorithm 
terminates [231 Theorem 4.7]. Although the efficiency of the line search algorithm depends on 
the parameters <5 and cr, thanks to the reference [331 Subsection 6.1], we can set appropriate 
<5 and a before executing it [331 Algorithm 4.6] and Algorithm 12.11 See Section [3] for the 
numerical performance of the line search algorithm [331 Algorithm 4.6] and Algorithm 12.II 
It can be seen that Algorithm 12.11 is well-defined when Pn is defined by /3®°, /3™, or 
• The discussion in Subsection 12.21 shows that Algorithm 12.11 with /3„ = is 
well-defined iLemma I2.3l ill. Moreover, it is guaranteed that under certain assumptions. 
Algorithm 12.II with /3„ = is well-defined (Theorem [33]). 


2.1 Algorithm 12.11 with jSn = 

This subsection considers Algorithm 12.11 with {n G N), which is based on the steepest 

descent (SD) direction (see (11.17^ 1. i.e., 

a;„+i := a;„-I-a„ (r(a;„) - a;„) {n € N). (2.5) 
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Theorems 4 and 8 in |24j indicate that, if (a„)neN satisfies the Armijo-type condition C3D, 
Algorithm (12.51) converges to a fixed point of T. The following theorem says that Algorithm 
(j2.5[l . with (an)ri6N satisfying the Wolfe-type conditions (EB and p.2ll . converges to a fixed 
point of T. 

Theorem 2.1. Suppose that (a;„)„gN the sequence generated by Alaorithm \2. 1\ with /3„ = 
(n S N). Then, {xn)n&i either terminates at a fixed point ofT or 

lim \\xn - T {Xn)\\ = 0. 

n—>oo 

In the latter situation, {Xn)nen weakly converges to a fixed point of T. 


2.1.1 Proof of Theorem 12.11 

If m G N exists such that \\xm — T{xm)\\ = 0, Theorem 12.11 holds. Accordingly, it can be 
assumed that, for all n G N, \\xn — T{xn)\\ ^ 0 holds. 

First, the following lemma can be proven by referring to [33 [331135] - 


Lemma 2.1. Let (dTt)nGN the sequences generated by Alaorithm \2.1\ Assume 

that {xn — T{xn),dn) < 0 for all n G N. Then, 



T{Xn),dn) 

\\du\\ 


2 

< OO. 


Proof. The Cauchy-Schwarz inequality and the triangle inequality ensure that, for all n G 
N, (d„, (Xn+l - TiXn+l)) - (Xn “ T{Xn))) < || dn || || (^n+l - T{Xn+l)) “ {Xn “ T{Xn))\\ < 
||d„||(||T(a:„) — T(a;„+i)|| -|- ||a;„+i — a;„||), which, together with the nonexpansivity of T and 
(El, implies that, for all n G N, 

{dn, {xn+i - T (a;„+i)) - {xn - T (xn))) < 2a„ \\dn\f ■ 


Moreover, (12.21) means that, for all n G N, 


{dn, {Xn+l - T (a;n-|-l)) - (Xn-T (Xn))) > (o’ “ 1) (dn, Xn - T (Xn)) ■ 


Accordingly, for all n G N, 

(cr - 1) {dn, Xn-T {Xn)) < ^Un \\dn\f ■ 

Since ||d„|| 7^ 0 (n G N) holds from (a;„ — T{xn), dn) < 0 (n G N), we find that, for all n G N, 

(cr - 1) {dn,Xn - T {Xn)) 


2||dn 


< an- 


( 2 . 6 ) 


Condition (EB means that, for all n G N, ||a:n-i-i — T{xn+i)\\'^ — \\xn — T{xn)\\'^ < San{xn — 
T{xn),dn), which, together with {xn — T{xn),dn) < 0 (n G N), implies that, for all n G N, 


Ctn < 


||cCn - T (ccn)ll^ - \\Xn+l - T {Xn+l)\\'^ 
d {Xn T {Xn) , dn) 


(2.7) 







From (12:61) and (12.71) . for all n G N, 


{a -l){dn,Xn-T{Xn)) ^ \\Xn - T {Xn)\\^ - \\Xn+l - T {Xn+l)\f 


ndn\? 

which implies that, for all n G N, 


d {Xn T (^Xn'j , dn} 


l5 (1 a'){dn,Xn T (x„)) ^ 11 ^ m|2 ||_ M|2 

2 — 11^" \^n)\\ |pn+l T {Xn+lj\\ ■ 

2 ||Mn|| 

Summing up this inequality from n = 0ton = A'^GN guarantees that, for all TV G N, 

f: < iix, - r(x,)ii» - iix„« - T 

< ||xo - T (xo)ll^ < OO. 

Therefore, the conclusion in Lemma [Q is satisfied. 

Lemma o leads to the following. 


□ 


Lemma 2.2. Suppose that the assumptions of Theorem \2.1\ are satisfied. Then, 

(i) lll^n—^oo \\Xn — 0 - 

(ii) {\\xn — a;||)„gN is monotone decreasing for all x G Fix(r). 

(hi) (a:n,)„gN weakly converges to a point in Fix(T). 

Items (i) and (hi) in Lemma 12 .2 1 indicate that Theorem 12.11 holds under the assumption 
that \\xn — T{xn)\\ 7 ^ 0 (n G N). 

Proof, (i) In the case where /3„ := = 0 (n G N), d„ = —(a;„ — T(a;„)) holds for all n G N. 

Hence, {xn — T{xn),dn) = —\\xn — r(x„)|p < 0 (n G N). Lemma [2d1 thus guarantees that 
Yj°^=o - T{x„)\\‘^ < OO, which implies lim„_>oo \\xn - T{xn)\\ = 0. 

(ii) The triangle inequality and the nonexpansivity of T ensure that, for all n G N and 
for all X G Fix(r), ||a;„+i-x|| = ||a:„ + a„(T(a;„)-x„)-x|| < (l-a„)||x„-x||+a„||r(x„)- 
T{x)\\ < ||x„ -x||. 

(hi) Lemma ET^ iil means that lim„_,.oo ||x„ — x|| exists for all x G Fix(T). Accordingly, 
( 2 ;rt)rt 6 N is bounded. Hence, there is a subsequence {xn,.)k^-M of (x„)„gr^ such that {xnk)k^n 
weakly converges to a point x* G H. Here, let us assume that x* ^ Fix(T). Then, Opial’s 
condition [281 Lemma 1], Lemma l2.2r i'). and the nonexpansivity of T guarantee that 

liminf ||x„fc - a;*|| < liminf ||x„,i^ - T (x*)|| 

k—¥c>o k—¥oc 

= liminf \\xn^ - T (x„J + T (x„J - T (x*)|| 

k—¥oc 

= liminf ||T(x„J -r(x*)|| 

k—¥oo 

< liminf ||x„,i^ - a^*||, 

k—^oo 

which is a contradiction. Hence, x* G Fix(T). Let us take another subsequence (Xnjig^ 
(C (x„)„gN) which weakly converges to x* G iL. A similar discussion to the one for obtaining 
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X* G Fix(T) ensures that x, G Fix(r). Assume that x* ^ x^. The existence of lim„_>oo 
x|| {x G Fix(T)) and Opial’s condition [281 Lemma 1] imply that 

lim \\xn - a;*|| = lim \\xn^ - a:* II < 1™ \\Xn^ - a;*|| 

n—>-cxD fe—fcx) k—¥oo 

= lim \\xn-x^,\\= lim ||a;„,-a:*|| 

n—¥oo i^co 

< lim ||a;„, - a;*|| = lim ||a:„ - a:*|| , 

i—^oo n—¥oc 

which is a contradiction. Therefore, x* = x*. Since any subsequence of (x„)„gN weakly 
converges to the same fixed point of T, it is guaranteed that the whole (x„)„gN weakly 
converges to a fixed point of T. This completes the proof. □ 

2.2 Algorithm 12.11 with jSn = 

The following is a convergence analysis of Algorithm 12.11 with /3„ = 

Theorem 2.2. Suppose that (x„)„gN is the sequence generated by Alaorithm \2.1\ with jSn = 
(n G N). Then, (xn)nGN either terminates at a fixed point ofT or 

lim \\xn - T (xn)ll = 0. 

n—>oo 

2.2.1 Proof of Theorem 12.21 

Since the existence of to G N such that ||xm — T{xm)\\ = 0 implies that Theorem 12.21 holds, 
it can be assumed that, for all n G N, ||x„ — T(x„)|| ^ 0 holds. Theorem 12.21 can be proven 
by using the ideas presented in the proof of |9l Theorem 3.3]. The proof of Theorem 12.21 is 
divided into three steps. 

Lemma 2.3. Suppose that the assumptions in Theorem A2.2\ a,re satisfied. Then, 

(i) {Xn - T{xn),dn) <0 (n G N). 

(ii) liminf„^oo \\xn - T{xn)\\ = 0. 

(hi) lim„^oo Ikn - T{xn)\\ = 0. 

Proof, (i) From do := -(xq - T(xo)), (xq - T{xo),do) = -||a:o - l’(a;o)|P < 0. Suppose 
that (xn — T{xn),dn) < 0 holds for some n G N. Accordingly, the definition of := 
{xn+i — T{xn+i)) — (x„ — T(xn)) and (12.21) ensure that 

{dm Vn) — {dm a^n+1 T (x^i-i-i)) {dji, Xn, T (x^^)) 

> (cr - 1) {dn, Xn-T (x„)) > 0, 

which implies that 

oDY ^ ||x„+i -r(x„+i)f 

■ {dmyu) ■ 

From the definition of := —(xn+i — T(a;n+i)) + we have 

(dn+li ^n+1 (^n+l)) — ||^n+l ^ (^n+l)|| “t“ /5yj {dn^ ^n-\-l ^ (^l^n+l)) 

= ||X „+1 - r(x„+i)f (-1 + (dmXn+l -T{Xn+,)) 

L {dn,yn) 

II rp / \ii2 {dm {Xn+1 ~ T {Xn+l)) ~ yn) 

= l|Xn+l -T(x„+i)|| - - -^; 

{dn,yn) 
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which, together with the definitions of yn and (> 0), implies that 


{dn+lj ^(^n+l)) — ||^n+l ^(^n+l)|| 


2 ^(^n)) 


{^n 5 yn) 


( 2 . 8 ) 


= (dn,X„ - T (Xn)) < 0. 


Induction shows that (x„ — T(x„),d„} < 0 for all n e N. This implies > 0 (n € N); 
i.e., Algorithm 12.II with /3„ = is well-defined. 

(ii) Assume that liminf„_>.oo Ha^n — '7"(2:n)|| > 0. Then, there exist no G N and e > 0 
such that ||a:n — T{xn)\\ > £ for all n > no. Since we have assumed that ||x„ — T(a;„)|| ^ 0 
(n G N), we may further assume that ||x„ — T(x„)|| > e for all n G N. From the definition 
of dn+i ■■= -{Xn+I - T{xn+i)) + Pn'^dn (n G N), we have, for all n G N, 

\\dnf = \\dn+l + (Xn+1 - T (x„+i))f 

= ||(i„+l||^ -I- 2 {dn+l,Xn+l - T {Xn+l)) + ||x„+i - T (x„+i)||^ . 

Lemma [231(1) and (j2.8p mean that, for all n G N, 


oDY _ {dn+lTXn+1 T i^Xn+l)) 
{dn,Xn - T (Xn)) 


Hence, for all n G N, 


IMn+lir 


{dn+1 ^ Xfi+l T(x72-|_i)) 

||x„+i - T (x„+i)||^ 


\K 


(d„+l,X„+i - r(x„+i))^ (dn-|-l,X„+i - r(x„+i)) {dn,Xn - T (Xn))^ 

|2 


lldr, 


1 


{dn,Xn-T{Xn))^ ||x„+i - T (x„+i) ||^ 

_ I _ ||x„+i-r(x„+i)|| Y 

||x„+i - T (x„+i)|| {dn+l,Xn+l - T (x„+i)) / 

2 


< 


\\d„ 


1 


(d„,x„ - r(x„))^ ||x„+i - T(x„+i)||^ 

Summing up this inequality from n = 0ton = A^GN yields, for all A^ G N, 

/ IMof 1 


< 


iV+1 

E 


(dAr+i,X7V-|-l - T(xAr+i))^ (do, Xq - T (xq))^ ||xfc - T (xfc) ||^ ’ 

which, which together with ||x„ — T(x„)|| > e (n G N) and do ■= —(xo — r(xo)), implies 
that, for all A^ G N, 




N-\-l 

^E 


1 


< 


N + 2 


{dN+l,XN+l - T {xN+l)) ||xfe - r(xfe)||^ 

Since Lemma |23Ki) implies ||d„|| ^ 0 (n G N), we have, for all A^ G N, 


(dw-i-i, xw-i-i - T (xAf +i))^ ^ 


p^+iir 


N + 2 
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Therefore, Lemma 12.11 guarantees that 


OO 

oo > ^ 


{dk,Xk - T {xk)) 

II4II 


> 


^ k + l 


= OO. 




This is a contradiction. Hence, liminf^-j-oo \\xn — ^(a^n)|| = 0. 

(iii) Condition (|2.ip and Lemma [TS^i) lead to that, for all n € N, 

||a;„+i - T (a;„+i)||^ - ||a;„ - T (a;„)||^ < Sun {x^ - T (a:„), d„) < 0. 


Accordingly, (||x„ — T(a;„)||)n,gN is monotone decreasing; i.e., there exists lim„^oo \\xn — 
T{xn)\\- Lemma lOT ii') thus ensures that lim„_>oo ||a;n — T{xn)\\ = 0. This completes the 
proof. □ 


2.3 Algorithm 12.11 with y = 

To establish the convergence of Algorithm [2T] when /3„ = , 8 ™, we assume that the step sizes 
an satisfy the strong Wolfe-type conditions, which are (ED and the following strengthened 
version of ( 12 . 21 ) : for a < 1 / 2 , 

\{Xn (a„) - T (Xn («„)) ,d„)| < -a(Xn - T {x„),dn) . (2.9) 

See [T] on the global convergence of the FR method for unconstrained optimization under 
the strong Wolfe conditions. 

The following is a convergence analysis of Algorithm 12.11 with /3„ = /3™. 

Theorem 2.3. Suppose that (a:„)„gN the sequence generated by Alaorithm \2.1\ with /3n = 
/3n^ {n G N), where {an)n^n satisfies (12.11) and (12.91) . Then (x„)„gN either terminates at a 
fixed point of T or 


lim \\xn 

n—>oo 


T{Xn)\\=0. 


2.3.1 Proof of Theorem 12.31 

It can be assumed that, for all n G N, \\xn — T(x„)|| 7 ^ 0 holds. Theorem 12.31 can be proven 
by using the ideas in the proof of [U Theorem 2]. 

Lemma 2.4. Suppose that the assumptions in Theorem \2.3\ are satisfied. Then, 

(i) {xn - T{Xn),dn) <0 (n G N). 

(ii) liminf„^oo \\xn - T{xn)\\ = 0 . 

(iii) lim„^oo \\xn - T{xn)\\ = 0 . 

Proof, (i) Let us show that, for all n G N, 


< 

1=0 


{Xn T (Xti) , dn) 
\\Xn - T {Xn)\\^ 


< —2 + afi 
1=0 


( 2 . 10 ) 
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From do := —(a;o — T{xo)), (12.101) holds for n := 0 and {xq — T{xo),do) < 0. Suppose 
that (|2.10|1 holds for some n S N. Accordingly, from X]j=o ~ 1/(1 ~ 

a S ( 0 , 1 / 2 ], we have 

{Xn - T (Xn) ,dn) ^ o , 7 “ (1 “ ^Cr) ^ ^ 

Ikn-'r(a7„)|| ^ 1-CT 

which implies that (a;„ — T{xn),dn) < 0. The definitions of d„+i and enable us to 
deduce that 

{Xn+l -T{Xn+l) ,dn+l) _ {xn+1 - T {Xn+l) , - {Xn+1 - T jXn+l)) + Pn^dn) 

||a:„+i - T (a;„+i)||^ ||a;„+i - T (a;„+i)||^ 

^ _|_ 11^71+1 ^ ((^'n+l)|| ((^'n+l T (Xn+l) 5 dri) 

||a:„ - r(a;„)||^ ||a:„+i - T (a;„+i)|l^ 

_ ^ ^ (^n +1 ^ (^n+l) ; ^n) 

||x„ - T {Xn)\\^ 

Since (12.9|) satisfies a{xn — T(a:„),d„) < {xn+i — T{xn+i),dn) < —cr{xn — T{xn),dn) and 
( 12 . 101 ) holds for some n, it is found that 

^ ^ (^n+l T (Xyj+i) , dri) ^ ^ ^ {Xn T (Xn) , dn) 

||x„-r(x„)||^ “ ||a:„ - r(xn)||^ 

n n +1 

> —1 — a (jl = — (jl 
1=0 1=0 


and 


^ ^ (Xn+l T (Xn+l) , dn) ^ ^ (^n) 5 d^^) 

||x„-T(Xn)||^ “ ||x„-T(x„)||^ 


< — 1 + cr cr-^ 
1=0 


n+1 


1=0 


Hence, 


n+1 

< 

1=0 


((^n+1 ^ (^n+l) ; ^n+l) 

||x„+i - T (x„+i)||^ 


n+1 

< -2 + ^( 

1=0 


A discussion similar to the one for obtaining (x„ — T(xn),dn) < 0 guarantees that (x„+i — 
T(x„+i),d„+i) < 0 holds. Induction thus shows that (12.101) and (x„ — T(x„),d„) < 0 hold 
for all n G N. 

(ii) Assume that liminf„_>oo ||a^n — T(x„)|| >0. A discussion similar to the one in the 
proof of Lemma [231 ii') ensures the existence of e > 0 such that ||xn — T(x„)|l > e for all 
n G N. From (|2.9p and (|2.10p . we have for all n G N, 


n+1 

I (^n+1 T , d-fi} I O' {xji T {xn) , df^ ^ ^ ^ o^ \\xji T (xyj) II , 
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which, together with YTjtl = ^/{^-^) and/3™ := \\xn+i-T{xn+iW/\\xn- 

T(x„)|p (n S N), implies that, for all n S N, 


,5™ \{xn+i - T {Xn+i),dn)\ < \\xn+i - T (a;„+i)||^ . 

1 — (7 


Accordingly, from the definition of dn+i ■= —{xn+i — T{xn+i)) + fi^dn, we find that, for 
all n G N, 


||d„+i II = Wn dn - {Xn+I “ T (a;„+i)) 


= ||d„r - 2/3^ {dn, Xn+i - T (x„+i)) + ||x„+i - T (a;„+i)|r 


< 


ll^ri+i - T (a;„+i)|| 


||Xn T {Xn 
which means that, for all n G N, 


II dr, 


2cr 

1 — a 


+ 


l) ||a:„+i - T (a;„+i)||^ , 


l|dn+iir 


< 


II dr. 


1 + cr 


||a;„+i - T(a;„+i)||'‘ ||x„ - T (a;„)||'‘ 1 “ ||x„+i - T (a;„+i)||^ 


The sum of this inequality from n = 0ton = A^GN and do := —{xq — T{xq)) ensure that, 
for all IV G N, 


||dAr+i 


< 


llxjv+i - T(a:Ar+i)||'‘ ll^o - T(xo)||^ 


1 + cr 
\ — a 


iV+l 

E 


^ ||xfc - T {xkW 


From ||a;„ — T(a:Ti)|| > e (n G N), for all A' G N, 

||djv+if ^ 1 ^ l + gA + l (l + tT)A + 2 

\\xN+i - T {xN+i)f ~ 

Therefore, from Lemma IT^ il guaranteeing that ||d„|| 0 (n G N) and ^^(1~^)/((1 + 

cr){k — 1) + 2) = 00 , it is found that 

||a;fe - T (a;fe)||'‘ 

h \\du\? 


Meanwhile, since (12.101) guarantees that {xn — T{xn),dn) < (—2 + ^J^q cr3)||x„ — r(x„)|p < 
(—(1 — 2cr)/(l — a))\\xn — T{xn)\\'^ (n G N), Lemma ITT] and Lemma VIM i] lead to the 
deduction that 


({xk ~ T ixk) , dfc) 

—w— 



lixfc - T (a;fc)||^ 
\\dkf 


= oo. 


which is a contradiction. Therefore, liminf„_>oo ||a^n — Tixn)\\ = 0. 

(iii) A discussion similar to the one in the proof of Lemma I2.3f iiil leads to Lemma 
I2.4r iiil. This completes the proof. □ 
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2.4 Algorithm 12.11 with Pn = 

It is well known that the convergence of the nonlinear conjugate gradient method with 
defined as in (11.191) for a general nonlinear function is uncertain [161 Section 5]. To guarantee 
the convergence of the PRP method for unconstrained optimization, the following modifica¬ 
tion of was presented in [3T]: for defined as in (11.191) . := max{,5™^, 0}. 

On the basis of the idea behind this modification, this subsection considers Algorithm 12.11 
with defined as in (12.41) . 

Theorem 2.4. Suppose that (a:„)„gN and (dnjngN o,re the sequences generated by Algorithm 
\2.1\ with Pn = (n G N) and there exists c > 0 such that (x„ — T{xn), dn) < —c||a;„ — 

T(x„)|p for all n £ N. If bounded, then (x„)„gN either terminates at a fixed 

point of T or 


lim ||xn - T (xn)ll = 0. 

n—^oo 

2.4.1 Proof of Theorem 12.41 

It can be assumed that ||x„ — T(x„)|| ^ 0 holds for all n G N. Let us first show the following 
lemma by referring to the proof of m Lemma 4.1]. 

Lemma 2.5. Let (x„)„gN (dn)neN be the sequences generated by Algorithm \ 2.1\ with 
/3„ > 0 (n £ N) and assume that there exists c > 0 such that (x„ — T{xn),dn) < —c||x„ — 
T(x„)|p for all n £ N. If there exists e > 0 such that ||x„ — T(x„)|| > e for all n £ N, then 
ll'“"+i “ where Un := dn/\\dn\\ (n £ N). 

Proof Assuming ||x„ - r(x„)|| > e and (x„ - T(x„),(i„) < -c||x„ - T(x„)|p (n £ N), 
||d„|| 0 holds for all n £ N. Define r„ := -(x„ - T{xn))/\\dn\\ and (5„ := /3„||d„||/||d„+i|| 

(n £ N). From SnUn = P„dn/\\dn+i\\ and dn+i = -(x„+i - r(x„+i)) + findn (n £ N), we 
have for all n € N, 


'^n+l — T"n-\-l 

which, together with ||m„+i - = ||m„+i||^ - 2(5„(m„+i,m„) -h (5^||w„|P = ||m„P - 

26n{un,Un+i) 4- 5l\\un+i\? = \\un “ SnUn+iW^ {ji £ N), implies that, for all n £ N, 

Ihn-t-lll — ll'^n+1 — ||^n ^n^n-t-l|t • 

Accordingly, the condition /3„ > 0 (n £ N) and the triangle inequality mean that, for all 
n £ N, 


||U„+1 - M„|| < (1 -I- S„) ||Mn+l - M„|| 

^ T W'^n (^41) 

= 2 ||r„+i|| . 


From Lemma ITTl (x„ — T{xn),dn) < —c||x„ — T(x„)|p (n £ N), the definition of rn, and 
||x„ — T{xn)\\ > £ (n £ N), we have 


f {Xn — T [Xn) , dn) \ ^ 2 

^ 


n=0 


n I 


-T{a 


n—0 


\K 


>cvj:\ 


n—0 


which, together with (I2.11L completes the proof. 


□ 
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The following property, referred to as Property (*), is a result of modifying m Property 
(*)] to conform to Problem (HU). 

Property (*). Suppose that there exist positive constants 7 and 7 such that 7 < ||a;„ — 
T{xn)\\ < 7 for all n G N. Then Property (*) holds if 6 > 1 and A > 0 exist such that, for 
all n G N, 

\Pn\<h and \\xn+i - a;„|| < A implies |/3„| < 

The proof of the following lemma can be omitted since it is similar to the proof of m 
Lemma 4.2]. 

Lemma 2.6. Let (x„)„gN and {dn)nen be the sequences generated by Algorithm \2.1\ and 
assume that there exist c > 0 and 7 > 0 such that (x„ — T{x„),dn) < —c||a::„ — T(x„)|p 
and \\xn — T{xn)\\ > 7 for all n G N. Suppose also that Property (*) holds. Then there 
exists A > 0 such that, for all A G N\{0} and for any index ko, there is k > ko such 
that > A/2, where ^ := {i G N\{0}: k<i<k + A — 1, ||a:i — > A} 

(fc G N, A G N\{0}, A > 0) and stands for the number of elements of 

The following can be proven by referring to the proof of HU Theorem 4.3]. 

Lemma 2.7. Let be the sequence generated by Alaorithm \2. 1\ with /3„ > 0 (n G N) 

and assume that there exists c > 0 such that (a;„ — T(xn),dn) < —cj|x„ — T{xn)\\'^ for all 
n G N and Property (*) holds. If {xn)n^n is bounded, liminf„_>oo \\xn — T(a::„)]| = 0. 

Proof. Assuming that liminf„^oo ||a:n — T(x„)\\ > 0, there exists 7 > 0 such that l|x„ — 
T{xn)\\ > 7 for all n G N. Since c > 0 exists such that {xn — T{xn), dn) < —cl|x„ — T(x„)ll^ 
(n G N), lldnl] 7 ^ 0 (n G N) holds. Moreover, the nonexpansivity of T ensures that, for 
all X G Fix(T), j|T(x„) — a;]] < l|x„ — xj], and this, together with the boundedness of 
(x„)„gN, implies the boundedness of {T(xn))nGn- Accordingly, 7 > 0 exists such that 
j|x„ — T{xn)\\ < 7 (u G N). The definition of x„ implies that, for all n > 1, 

Xn Xn — 1 — ^n — ldn — 1 — ^n — 1 ||dn — 1 || Un — 1 — \\Xn a^n— 1 || Un—l, 
where Un '■= dn/Udnll {n G N). Hence, for all l,k gN with l> k > 0, 

i i 

XI - Xk-l = Xi-l) = ^ IJxi - Xi-lJI Ui-l, 

i—k i—k 


which implies that 

i i 

\\Xi - Xi_il] Uk-l = Xl- Xk-l - ||Xi - Xi_il] {Ui-I - Uk-l) ■ 
i—k i—k 

From l]wnjj = 1 (n G N) and the triangle inequality, for all /, fc G N with I > k > 0, 

Yl\=k ll^i “ Xi-i\\ < \\xi - Xk-i\\ + Yl\=k ll^i “ a^i-illllMi-i - ■Wfe-ill. Since the boundedness 
of (x„)„gN means there is M > 0 satisfying l|x„+i — x„ll < M (n G N), we find that, for all 
l,k gN with I > k > 0, 


i i 

iixi - Xi-iji < M+ jjxi - Xi-i\\ \\ui-i - Mfe-iii. 
i—k i—k 


( 2 . 12 ) 
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Let A > 0 be as given by Lemma [2^ and define A := [dM/A], where f-] denotes the 
ceiling operator. From Lemma 12.51 an index fco can be chosen such that ~ 

Ui-iW^ < 1/(4A). Accordingly, Lemma EH guarantees the existence of fc > fco such that 
^ SiiiC6 ttiG C3/iictiy Scliw3,rz iiiGQU&-lity impliGS ^ ^ —-i cii'^ ^ tti ^ —i 

(m > 1, € M, z = 1, 2,..., m), we have, for alH € [k,k A — 1], 


2-1 


2-1 






j=k 


Putting / := fc + A — 1, (12.121) ensures that 

fe+A-l 

2 


.. fc+A — 1 

M>- ol^fe.A 


> 


AA 


i—k 



which implies that A < 4M/A. This contradicts A := \AM/X\. Therefore, liminf„_>oo \\xn — 
T{Xn)\\=0. □ 


Now we are in the position to prove Theorem 12.41 


Proof. The condition > 0 holds for all n G N. Suppose that positive constants 7 and 

7 exist such that 7 < \\xn — T{xn)\\ < 7 (n G N) and define b := 27 ^/ 7 ^ and A := 7 ^/( 47 &). 
The definition of and the Cauchy-Schwarz inequality mean that, for all n G N, 

I oPRP+l ^ I {Xn+1 — T (Xn+l) , Un) \ ^ ll^n+l ~ ^(^ra+l)ll WUnW ^ ^7^ ^ ^ 
\\xn-T{xn)\\^ ||a;„ - T (a:„)||^ 7^ 

where the third inequality comes from ||j/„|| < ||a:n,+i — T{xn+i)\\ + \\xn — T{xn)\\ < 27 and 
7 < \\xn — T{xn)\\ < 7 (u G N). When |la;„+i — x„|| < A (n G N), the triangle inequality 
and the nonexpansivity of T imply that ||y„|| < ||x„+i — Xn\\ + \\T(xn) — P(a^rt+i)|| < 
2||a;„+i — Xn\\ < 2A (n G N). Therefore, for all n G N, 

I oPRP+ I ^ 7 WUnW _ J_ 

||x„-T(x„)f - 7 ^ 26’ 

which implies that Property (*) holds. Lemma |2.7I thus guarantees that liminf„_>.oo \\xn — 
T{xn)\\ = 0 holds. A discussion in the same manner as in the proof of Lemma l2.3l iiil leads 
to lim„^oo ||a^rt — T{xn)\\ = 0. This completes the proof. □ 


2.5 Algorithm 12.11 with 

The convergence properties of the nonlinear conjugate gradient method with defined as 
in (11.191) are similar to those with defined as in (11.191) [l6l Section 5]. On the basis of 
this fact and the modification of in Subsection l2.41 this subsection considers Algorithm 
12 .II with defined by (12. 4p . 

Lemma E71 leads to the following. 

Theorem 2.5. Suppose that (a:„)„gN and {dn)nGn are the sequences generated by Algorithm 
\2.1\ with jdn = {'O' G N) and there exists c > 0 such that {xn — T{xn), dn) < —c\\xn — 

T(xn)\\'^ for all n G N. If (a;„)„gN bounded, then {xn)ne'M either terminates at a fixed 
point of T or 

lim \\xn - T {Xn)\\ = 0. 
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Proof. When m G N exists such that \\xm — T{xm)\\ = 0, Theorem [2^ holds. Let us consider 
the case where \\xn — T{xn)\\ ^ 0 for all n G N. Suppose that 7,7 > 0 exist such that 
7 < ||a;rt — T{xn)\\ < 7 (n G N) and define b := 27 ^/((l — cr)c 7 ^) and A := (1 — ct)c7^/(47&). 
Then (j2.2l) implies that, for all n G N, 

yrt) ~ ^n+1 ^ (^n+l)) ^ (^n)) 

^ (1 (dn, Xn T (^Xyif) , 

which, together with the existence ofc ,7 > 0 such that (a;„ — T(a;„), d„) < —c||x„ —T(a::„)||^, 
and 7 < \\xn — T{xn)\\ {n G N), implies that, for all n G N, 

{dn,yn) > (1 - cr) C ||a;„ - T {Xn)\f > (1 - O') C 7 ^ > 0. 

This means Algorithm 12.11 with /3„ = is well-defined. From \\xn — T{xn)\\ < 7 (n G N) 
and the definition of j/„, we have, for all n G N, 

I om +1 < \{Xn+l -T{Xn+l),yn)\ ^ , 

\{dn,yn)\ 

When ||x„+i — Xn\\ < A (n G N), the triangle inequality and the nonexpansivity of T imply 
that \\yn\\ < \\xn+i - a;„|| -f ||T(x„) - T{xn+i)\\ < 2||x„+i - x„|| < 2A (n G N). Therefore, 
from \\xn — T{xn)\\ <jinG N), for all n G N, 

|oHS+| < ihnW , 2 A 7 ^ J_ 

" “ {dn,yn) ~ {l-cr)c'y^ 26’ 

which in turn implies that Property (*) holds. Lemma [2 . TI thus ensures that liminfji_>oo ||a;rt~ 
T{xn)\\ = 0 holds. A discussion similar to the one in the proof of Lemma [2.3r iiil leads to 
lim„_>oo ||a^rt — 'L(a;„)|| = 0. This completes the proof. □ 


2.6 Convergence rate analyses of Algorithm 12.1 

Subsections I2.1H2.5I show that Algorithm 12.11 with formulas (12.4p satisfies lim„_>oo Ha^n — 
T(xn)\\ = 0 under certain assumptions. The next theorem establishes rates of convergence 
for Algorithm 12.II with formulas (12.41) . 

Theorem 2.6. (i) Under the Wolfe-type conditions (EU and (12.21) . Alaorithm \2.1\ with 

fin = satisfies, for all n G N, 


\\Xn 


T{Xn)\\< 


||a:o-r(a;o)|| 

\/dSfe=0 


(ii) Under the strong Wolfe-type conditions and (12.91) . Alaorithm \2.1\ with fin = Pn'^ 
satisfies, for all n G N, 

||xo -r(a;o)|| 


-Tia 


< 




(hi) Under the strong Wolfe-type conditions ([231) and (ESI), Algorithm\K^with Pn = Pn^ 
satisfies, for all n G N, 


||a;n - T (a;„)|| < 


||xo-T(xo)|| 


\/ ELo (1 - 2a + fT'=) Ofc 
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(iv) Under the assumptions in Theorem \2.4\ Alaorithm \2. 1\ with fdn = satisfies, for 

all n G N, 


\\Xn - T {Xn)\\ < 


||a:o-T(a;o)|| 

\/c<5ELo 


(v) Under the assumptions in Theorem \2.5l Alaorithm \2.1\ with /?„ = satisfies, for 

all n G N, 


\\xn - T (a:„)|| < 


lla^o -T(a;o)|| 

\/ c5 Y.k=o 


Proof, (i) From dk = —{xk — T{xk)) {k £ N) and (|2.1I) . we have that 0 < dak\\xk — T{xk)\\'^ < 
\\xk — T{xk)\\^ — ||a:fe+i — T{xk+i)\\‘^ {k £ N). Summing up this inequality from fc = 0 to 
k = n guarantees that, for all n £ N, 

n 

\\xk - T {xk)\\^ < ||cco - T (a;o)||^ - ||a;„+i - T (a;n+i)||^ < ||a;o - T (a;o)||^ , 

k=0 

which, together with the monotone decreasing property of (||a;„ — T(a:n)||^)ngN, implies that, 
for all n £ N, 


S \\xn - T (a;„)||^ ^ a* < ||a;o - T (xo)||" 


k=0 


This completes the proof. 

(ii) Condition (1^ and Lemma i) ensure that —a < {xk+i ~ T{xk+i),dk)/{xk — 
T{xk), dk) < a {k € N). Accordingly, (j2.8|) means that, for all fc £ N, 


{^k+l ^ — 


{xk - T {xk),dk) 


{dkiixk+l U’ (^Xk-\-l)) {Xk ^ (.^k))) 


\\xk+i - T (xfe+i) 


1 + cr 

Hence, (EH) implies that, for all fc £ N, 


/ {xk+i - T{xk+i),dk) 

V {xk - T{xk),dk) 

< - — \\xk+i - T (a;fc+i)||^ . 


-1 ||a:fc+i - T(a;fc+i)||" 


||a;fc+i - T (a;fc+i)|| - - T {xk)\\ < “T";— \\xk - T {xk)\\ ■ 

1 + cr 

Summing up this inequality from fc = 0 to fc = n and the monotone decreasing property of 
{\\xn — T(a;n)|p)ngN ensure that, for all n £ N, 


1 


1 + cr 


5 \\xn - T (a:„)||^ '^ak < ||a;o - T (xo)!!^ , 


k=0 


which completes the proof. 

(iii) Inequality (12.101) guarantees that, for all fc £ N, 


{xk - T {xk),dk) < ( “2 + ^cr-^ 1 \\xk - T (a;fe)||^ = --— ^ ^ ||a:fe - T {xk)"‘^ 


3=0 


— a 
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which, together with (12.11) . implies that, for all k G N, 


lixfe+i - T (a;fe+i)||^ - \\xk - T {xk)\\^ < - - - ~^^^-^5ak \\xk - T {xk)\f ■ 

1 — a 

Summing up this inequality from k = 0 to k = n and the monotone decreasing property of 
{\\xn — T(a;n)|P)neN ensure that, for all n G N, 

1 ” 

-- S \\xn - T (a;„)||^ ^ (l - 2cr + cr^) ak < ||a;o - T (xo)|l^ , 

1 — cr 


which completes the proof. 

(iv), (v) Since there exists c > 0 such that {xk — T{xk),dk) < —c\\xk — T{xk)\\^ for all 
fc G N, we have from (EB and the monotone decreasing property of (||a;n — T’(a;„)|p)ngN 
that, for all n G N, 


c6 ||xn - T (xn)ll^ ^ ak \\xk - T {xk)\f < \\xo - T (xo)||^ . 

A;=0 fc=0 


This concludes the proof. 


□ 


The conventional Krasnosel’skii-Mann algorithm (EH) with a step size sequence (a„)„gN 
obeying (lESl) satisfies the following inequality [H Propositions 10 and 11]: 


\\Xn 


T{Xn)\\ < 


d(xo,Fix(r)) 

V'^k=0 


(n G N), 


where d(xo, Fix(T)) := min 2 ;gFix(T) ||a^o ~ a:||. When a„ (n G N) is a constant in the range of 
(0,1), which is the most tractable choice of step size satisfying (II.3L the Krasnosel’skii-Mann 
algorithm (EE has the rate of convergence, 


\\xn-T{xn)\\=o(^-j^y (2.13) 

Meanwhile, according to Theorem 5 in [24], Algorithm (11.21) with (Q;„)„gN satisfying the 
Armijo-type condition EE satisfies, for all n G N, 


ll^^n 


T{Xn)\\< 


||xo -T(xo)|| 

\//3ELo K-l)' 


(2.14) 


In general, the step sizes satisfying EE do not coincide with those satisfying the Armijo- 
type condition EE or the Wolfe-type conditions EE and (1^ . This is because the line 
search methods based on the Armijo-type conditions (11.51) and (12.11) determine step sizes at 
each iteration n so as to satisfy ||x„+i — T(xn+i)\\ < ||x„ — T(x„)||, while the constant step 
sizes satisfying (11.31) do not change at each iteration. Accordingly, it would be difficult to 
evaluate the efficiency of these algorithms by using only the theoretical convergence rates 
in ()2.13L (I2.14L and Theorem 12.61 To verify whether Algorithm 12.11 with the convergence 
rates in Theorem 12.61 converges faster than the previous algorithms jS] Propositions 10 and 
11], |2U Theorem 5] with convergence rates ()2.13l) and (I2.14p . the next section numerically 
compares their abilities to solve concrete constrained smooth convex optimization problems. 
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3 Application of Algorithm 12.11 to constrained 
smooth convex optimization 

This section considers the following problem: 

Minimize / {x) subject to x £ C, (3.1) 

where f: K is convex, V/: —>■ is Lipschitz continuous with a constant L, 

and C C is a nonempty, closed convex set onto which the metric projection Pq can be 
efficiently computed. 


3.1 Experimental conditions and fixed point and line 
search algorithms used in the experiment 

Problem can be solved by using the conventional Krasnosel’skii-Mann algorithm (HU 
with a nonexpansive mapping T := Pc(Id — AV/) satisfying Fix(T) = argmin,j, 
where A G (0, 2/L] [191 Proposition 2.3]. It is represented as follows: 

Xn-\-l — Xfi O-ji {Pc (^n AV/ (Xt^)) , (^•^) 

where a:o G and (a„)„gN is a sequence satisfying (11.31) or the Armijo-type condition (11.51) . 
Algorithm 12.11 with T := Pc (Id — AV/) is as follows: 


dn-t -1 i^n+1 Pc (^n-t -1 AV/ (x^i-i-i))) Pndn^ 


where Xo,do := — (ccq — Pc(a^o — AV/(a:o))) G R'^, (Q!n)nGN is a sequence satisfying the Wolfe- 
type conditions and (|2.2L and (/l„)„gN is defined by each of the formulas (12.41) with 
T := Pc(Id - AV/) (see also (fmi) l. 

The best conventional nonlinear conjugate gradient method for unconstrained smooth 
nonconvex optimization was proposed by Hager and Zhang [141 HSli and it uses the HS 
formula defined as in (|1.19l) : 


:= 


1 


{drii Vn) 


Un ‘2dn 


hn 


{dniVn) 


,V/(xn+i) 


g hnf (V/(^n+l),dn) 
" {dn,yn) {dn,yn) 


Replacing V/ in the above formula with Id — Pc (Id — AV/) leads to the HZ-type formula 
for Problem dlD: 

qHZ oHS O lly"ll (^^n+l ~ Pc ( 2 : 71+1 ~ AV/ (Xn+l)) , dn) 7.5 ^ 

Pn ■ Pn \ /+ \ > la.aj 

{d„,yn) {dn,yn) 

where yn ■= (xn+i - Pc(a;„+i - AV/(a;„+i))) - (x„ - Pc(a;„ - AV/(a;„))) andis defined 
by := {xn+i - Pc{xn+i - AV/(a:„+i)),i/„)/(d„, j/„/ We tested Algorithm ([331) with 
jin := defined by (|3.4I) in order to see how it works on Problem (|3.1I) . 

We used the Virtual Desktop PC at the Ikuta campus of Meiji University. The PC has 8 
GB of RAM memory, 1 core Intel Xeon 2.6 GHz GPU, and a Windows 8.1 operating system. 
The algorithms used in the experiment were written in MATLAB (R2013b), and they are 
summarized as follows. 
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SD-1: Algorithm (13.21) with constant step sizes a„ := 0.5 (n G N) [3l Theorem 5.14]. 

SD-2: Algorithm (13.21) with a„ satisfying the Armijo-type condition (11.51) when /3 = 0.5 
[24l Theorems 4 and 8]. 

SD-3: Algorithm p.3p with a„ satisfying the Wolfe-type conditions (12.11) and (12.21) and 
fin '■= (Theorem 12.II) . 

FR: Algorithm (13.3p with satisfying the Wolfe-type conditions (12.11) and (12.21) and /3„ := 

/3™ (Theorem 12.3p . 

PRP-|-: Algorithm (13.31) with satisfying the Wolfe-type conditions (12.11) and (12.21) and 
Pn '■= (Theorem 12.41) . 

HS-|-: Algorithm (13.31) with satisfying the Wolfe-type conditions (12.1|) and (12.21) and 
Pn '■= (Theorem 12.5|) . 

DY: Algorithm (|3.3I) with an satisfying the Wolfe-type conditions ^2TD and (1^ and 
Pn ■■= Pn'^ (Theorem 12 .21) . 

HZ: Algorithm (13.31) with satisfying the Wolfe-type conditions (12.11) and (12.21) and /3„ := 
P™ defined by (13.41) [TTl ITS] . 

The experiment used the following line search algorithm Algorithm 4.6] to find step 
sizes satisfying the Wolfe-type conditions and (1^ with S := 0.3 and a := 0.5 that 
were chosen by referring to [23l Subsection 6.1], where, for each n, A„(-) and Wn(-) are 

An{t) : ||a:„ (t) - T (a;„ (f))]]^ - ||a:„ - T {xn)f < St (x„ - T (x„), (i„), 

Wnit) : {Xn (t) - T {Xn (()) , dn) > d (x„ - T (x„) , dn) ■ 

Algorithm 3.1. [531 Algorithm 4.6] 

Require: An{-),Wn{-). 

Ensure: A„(a) and Wn{a). 
a i — 0, P i — oo, t i — 1. 

loop 

if then 

P ^ t. 

else if ->Wn(t) then 
a t. 
else 

(a : found). 

end if 

if P < oo then 

t i — ^(O! -|- P). 
else 

t i — 2o. 

end if 
end loop 

For Algorithm SD-2, we replaced A„(-) above by 

An{t) : gn (() - 9ni0) < -Dt ||x„ - T {Xn)\f , 
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where D := S = 0.3 and gn is defined as in (HD, and deleted Wni-) from the line search 
algorithm. For Algorithms FR, PRP+, HS+, DY, and HZ, if the step sizes satisfying the 
Wolfe-type conditions HD and p.2p were not computed by using Algorithm 13.11 the step 
sizes were computed by using Algorithm 13.11 when dn ■= —(xn — T(xn)). This is because 
Algorithm 13.II for Algorithm SD-3, which uses := —(xn — T(xn)) (n G N), had a 100% 
success rate in computing the step sizes satisfying (HD and (| 2 . 2 |) . Tables [TJ [ 2 l [3l and |4] 
indicate the satisfiability rates (defined below) of computing the step sizes for the algorithms 
in the experiment. 

The stopping condition was 

n = 10 or \\Xno — T (a;no)ll = 0 for some no G [0,10]. (3.5) 

Before describing the results, let us describe the notation used to verify the numerical 
performance of the algorithms. 

• I: the number of initial points 

• Xq^ : the initial point chosen randomly (i = 1,2,... ,1) 

• ALGO: each of Algorithms SD-1, SD-2, SD-3, FR, PRP+, RS+, DY, and HZ 
(ALGO e {SD-1, SD-2, SD-3, FR, PRP+, HS-H, DY, HZ}) 

• iVi(a;Q*\ ALGO): the number of step sizes computed by Algorithm 13.11 for ALGO 
with Xq^ before ALGO satisfies the stopping condition (13.51) 

• iV 2 (a:Q*\ ALGO): the number of iterations needed to satisfy the stopping condition 
(13.51) for ALGO with Xg'^ 

Note that Ni(xq\ SD-1) stands for the number of iterations n satisfying A„(0.5) and 
VF„(0.5) before Algorithm SD-1 with Xg^ satisfies the stopping condition (13.51) . The sat¬ 
isfiability rate (SR) of Algorithm 13.II to compute the step sizes for each of the algorithms is 
defined by 


SR(ALGO) := 


SL^i(4^algo) 

Ell algo) 


X 100 [%]. 


(3.6) 


We performed 100 samplings, each starting from different random initial points (i.e., I := 
100 ) and averaged their results. 


3.2 Constrained quadratic programming problem 

In this subsection, let us consider the following constrained quadratic programming problem: 

Problem 3.1. Suppose that C is a nonempty, closed convex subset ofW^ onto which Pc 
can he efficiently computed, Q G is positive semidefinite with the eigenvalues Amin := 

Ai, A 2 , ... ,Xd =■ Amax satisfying Xi < Xj (i < j), and b G Our objective is to 

minimize f{x) := - {x, Qx) + {b, x) subject to x G C. 
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Since / above is convex and V/(a;) = Qx + b {x G is Lipschitz continuous such 
that the Lipschitz constant of V/ is the maximum eigenvalue Amax of Q, Problem 13.II is an 
example of Problem (j3.1l) . 

We compared the proposed algorithms SD-3, FR, PRP+, HS+, DY, and HZ with the 
previous algorithms SD-1 and SD-2 by applying them to Problem 13.11 li.e.. the fixed point 
problem for T{x) := Pc{x — (2/Xynax){Qx + b)) {x G R'^)) in the following cases: 

d := 10^ or 10'‘, Amin := 0, Amax := d, X, G [0, d] (i = 2, 3,..., d - 1), 
b,cG {-32,32f, C:= {xGR‘^: ||x - c|| < l} . 

We randomly chose Xi G [0,d] {i = 2,3,... ,d — 1) and set Q as a diagonal matrix with 
eigenvalues Ai, A 2 ,..., Amax- The experiment used two random numbers in the range of 
(—32, 32)'^ for b and c to satisfy C fl {x S : V/(x) = 0} = 0. Since C is a closed ball with 
center c and radius 1, Pq can be computed within a finite number of arithmetic operations. 
More precisely, Pc{x) := c + (x — c)/||x — c|l if ||x — c|| > 1, or Pc{x) := x if ||x — c|| < 1. 

Table 1: Satisfiability rate of Algorithm 13.11 for Algorithms SD-1, SD-2, and 
SD-3 applied to Problem l3 .1 1 when d := 10^, 10^ 


Algorithm 

SR (d := 10=^) 

SR {d := 10^) 

SD-1 

55.9% 

26.3% 

SD-2 

100% 

100% 

SD-3 

100% 

100% 


Table 2: Satisfiability rate of Algorithm 13.11 for Algorithms FR, PRP-I-, HS-I-, 
DY, and HZ applied to Problem 13.11 when d := 10^, 10^ 


Algorithm 

SR {d := 10^) 

SR {d := 10^) 

FR 

19.7% 

28.1% 

PRP+ 

100% 

100% 

HS+ 

100% 

98.9% 

DY 

21.6% 

27.2% 

HZ 

20.0% 

20.0% 


Table [U shows the satisfiability rates as defined by (13.611 for Algorithms SD-1, SD-2, and 
SD-3 that are applied to Problem 13. 11 It can be seen that the step sizes for SD-1 (constant 
step sizes an '■= 0.5) do not always satisfy the Wolfe-type conditions (12.11) and (12.211 . whereas 
the step sizes computed by Algorithm [HT] and SD-2 (resp. Algorithm SD-3) definitely satisfy 
the Armijo-type condition (11.51) (resp. the Wolfe-type conditions (12.11) and (12.21) 1. 

Table [2] showing the satisfiability rates for Algorithms FR, PRP-I-, HS-f, DY, and HZ 
indicates that Algorithm 13 .1 1 for PRP-I- and HS-I- has high success rates at computing the 
step sizes satisfying (ED) and p.2l) . while the SRs of Algorithm 13.II for other algorithms are 
low. It can be seen from Tables [T] and [5] that SD-3, PRP-I-, and HS-f are robust in the sense 
that Algorithm [3T] can compute the step sizes satisfying the Wolfe-type conditions (EB) and 

(IQ) . 
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Figure [T] indicates the behaviors of SD-1, SD-2, and SD-3 when d := 10^. The y-axes 
in Figures [UJa) and [Il[b) represent the value of \\xn — T{xn)\\- The x-axis in Figure [Il[a) 
represents the number of iterations, and the x-axis in Figure[IJb) represents the elapsed time. 
If the (||a;„ — T(x„)||)neN generated by the algorithms converges to 0, they also converge to 
a fixed point of T. Figure [IJa) shows that SD-2 and SD-3 terminate at fixed points of T 
within a finite number of iterations. It can be seen from Figure [Ija) and Figure [Ijb) that 
SD-3 reduces the iterations and running time needed to find a fixed point compared with 
SD-2. These figures also show that (||a;„ — T(a;„)||)jigN generated by SD-1 converges slowest 
and that SD-1 cannot find a fixed point of T before the tenth iteration. We can thus see 
that the use of the step sizes satisfying the Wolfe-type conditions is a good way to solve 
fixed point problems by using the Krasnosel’skh-Mann algorithm. Figure [5] indicates the 
behaviors of SD-1, SD-2, and SD-3 when d := 10'^. Similarly to what is shown in Figure [U 
SD-3 finds a fixed point of T faster than SD-1 and SD-2 can. 


10 ° 

lo'* 


10 ° 


= 10 ° 
^ 10 ^ 


10 “ 


10 ‘ 
10 '' 
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(a) \\xn — T{xn)\\ vs. no. of itera- (b) \\xn — T(x„)|| vs. elapsed time 
tions 


Figure 1: Evaluation of ||x„ — T(a;„)|| in terms of the number of iterations 
and elapsed time for Algorithms SD-1, SD-2, and SD-3 for Problem 13.11 when 
d := 10^ 




Elapsed time [sec] 


(a) ||x,i — r(x„)|| VS. no. of itera- (b) \\xn — T{xn)\\ vs. elapsed time 
tions 


Figure 2: Evaluation of ||xn — T{xn)\\ in terms of the number of iterations 
and elapsed time for Algorithms SD-1, SD-2, and SD-3 for Problem 13.11 when 


d := W 


25 







































































Figure [3] is the evaluation of (||x„ — T(a;„)||)„gN in terms of the number of iterations 
and elapsed time for Algorithms FR, PRP+, HS+, DY, and HZ when d := 10^. Figure 
[Hl^a) shows that they can hnd fixed points of T within a finite number of iterations. Figure 
[H^b) indicates that PRP+ and HS+ find the fixed points of T faster than FR, DY, and 
HZ. This is because Algorithm 13.11 for each of PRP+ and HS+ has a 100 % success rate 
at computing the step sizes satisfying m and (12.21) . while the SRs of Algorithm 13.11 for 
FR, DY, and HZ are low (see Tabled; i.e., FR, DY, and HZ require much more time to 
compute the step sizes than PRP+ and HS+. In fact, we checked that the times to compute 
the step sizes for FR, DY, and HZ account for 92.672202%, 87.156303%, and 83.700936% of 
all the computational times, while the times to compute the step sizes for PRP+ and HS+ 
account for 60.725204% and 60.889635% of all the computational times. Figure 0] indicate 
the behaviors of FR, PRP+, HS+, DY, and HZ when d := 10^ and PRP+ and HS+ perform 
better than FR, DY, and HZ, as seen in Figure [H Such a trend can be also verified from 
Table [2] showing that the SRs of Algorithm 13.II for PRP+ and HS+ are about 100%. 
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(a) \\Xn 
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— T{xn)\\ VS. no. of itera- (b) \\xn — T{xn)\\ vs. elapsed time 


Figure 3: Evaluation of ||a;„ — T{xn)\\ in terms of the number of iterations and 
elapsed time for Algorithms FR, PRP+, HS+, DY, and HZ for Problem I3.1I 
when d := 10^ 
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(a) \\xn — T{xn)\\ VS. no. of itera- (b) \\xn — T(x„)|| vs. elapsed time 
tions 


Figure 4: Evaluation of ||xn — T{xn)\\ in terms of the number of iterations and 
elapsed time for Algorithms FR, PRP+, HS+, DY, and HZ for Problem 13.11 
when d := 10^ 

3.3 Generalized convex feasibility problem 

This subsection considers the following generalized convex feasibility problem [6l Section I, 
Framework 2], Subsection 2.2], [371 Definition 4.1]: 

Problem 3.2. Suppose that Ci (i = 0,1,...,m) is a nonempty, elosed convex subset of 
onto which Pc^ can he efficiently computed and define the weighted mean square value 
of the distances from x G to Ci {i = 1,2, ...,m) as f{x) below; i.e., for Wi G (0,1) 
(i = 1, 2,..., m) satisfying YaLi 

m / >..2 

/(a;) ■ 

Our objective is to find a point in the generalized convex feasible set defined by 

Cf ■■= \x* &Co-. f (x*) = min / (a;)l . 

xGCo J 

C/ is a subset of Cg having the elements closest to Ci (i = 1, 2,..., m) in terms of the 
weighted mean square norm. Even if Hido ~ well-defined because C/ is the set 

of all minimizers of / over Cq. The condition C/ 0 holds when Co is bounded [371 Remark 
4.3(a)]. Moreover, C/ = HZlo holds when ^ Accordingly, Problem 13.21 is a 

generalization of the convex feasibility problem [2] of finding a point in Hldo C'i yf 0. 

The convex function / in Problem 13.21 satisfies V/ = Id — Hence, V/ 

is Lipschitz continuous when its Lipschitz constant is two. This means Problem 13.21 is an 
example of Problem (|3.1I) . Since Problem 13.21 can be expressed as the problem of finding a 
fixed point of T = Pco(Id ~ ^Y/) = Pco(Id ~ ^(Id ~ Sti WiPci)) ^ £ (Oj 1]> i^s^d 
T with A = 1; i.e., T := PcgiYiLi mPCi)- 

We applied SD-1, SD-2, SD-3, FR, PRP-f, HS-f, DY, and HZ to Problem 13.21 in the 
following cases: 

d := 10^ or 10"^, m := 99, ^ (* = 1; 2,... , 99), 

a G (-32, 32)^^, C, := {xGM'^: \\x - c,\\ < l} (i = 0,1,..., m). 
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The experiment used one hundred random numbers in the range of (—32, 32)^ for Ci, which 
means ~ Since Ci (* = 0,1,..., m) is a closed ball with center a and radius 1, 

Pi can be computed within a finite number of arithmetic operations. 


Table 3: Satisfiability rate of Algorithm 13.11 for Algorithms SD-1, SD-2, and 
SD-3 applied to Problem 13.21 when d := 10^, 10^ 


Algorithm 

SR {d := 10^) 

SR {d := 10^) 

SD-1 

80.6% 

64.2% 

SD-2 

100% 

100% 

SD-3 

100% 

100% 


Table 4: Satisfiability rate of Algorithm 13.11 for Algorithms FR, PRP+, HS+, 
DY, and HZ applied to Problem 13.21 when d := 10^, 10"^ 


Algorithm 

SR (d := 10=^) 

SR {d := 10^) 

FR 

50.0% 

50.0% 

PRP+ 

100% 

100% 

HS+ 

55.8% 

60.4% 

DY 

50.0% 

50.0% 

HZ 

50.0% 

50.0% 


Table [3] shows the satisfiability rates as defined by (13.6p for Algorithms SD-1, SD-2, and 
SD-3 applied to Problem 1 3.2 1 It can be seen that the step sizes for SD-1 do not always satisfy 
the Wolfe-type conditions and (12.21) . whereas the step sizes computed by Algorithm 13.II 
and SD-2 (resp. Algorithm SD-3) definitely satisfy the Armijo-type condition ()1.5|) (resp. 
the Wolfe-type conditions (EU and (12.21) 1. Such a trend also existed when SD-1, SD-2, and 
SD-3 were applied to Problem 13.II ('see Tabled]). 

Tabledjshows the satisfiability rates for Algorithms FR, PRP-f, HS-f, DY, and HZ. The 
table indicates that Algorithm 13.II for PRP-I- has a 100% success rate at computing the step 
sizes satisfying m and (lO) . while the SRs of Algorithm 13.11 for the other algorithms lie 
between 50% and about 60%. From Tables [3| and HJ we can see that SD-3 and PRP-f are 
robust in the sense that Algorithm 13.11 can compute the step sizes satisfying the Wolfe-type 
conditions ^2T| and (IT^ . 
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Figure 5: Evaluation of ||xn — T{xn)\\ in terms of the number of iterations 
and elapsed time for Algorithms SD-1, SD-2, and SD-3 for Problem 13.21 when 
d := 10^ 
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Figure 6: Evaluation of ^Xn — T{xn)\\ in terms of the number of iterations 
and elapsed time for Algorithms SD-1, SD-2, and SD-3 for Problem 13.21 when 
d := 10* 


Figure [5] indicates the behaviors of SD-1, SD-2, and SD-3 when d := 10^. The y-axes 
represent the value of \\Xn — T(x„)||. The x-axis in Figure [5fa) represents the number of 
iterations, and the x-axis in Figure [Sib) represents the elapsed time. From Figure jSfa), 
the iterations needed to satisfy ||a;„ — T{xn)\\ = 0 for SD-2 and SD-3 are, respectively, 3 
and 2. It can be seen that SD-3 reduces the running time and iterations needed to find 
a fixed point compared with SD-2. These figures also show that the (||a;„ — r(x„)||)„gN 
generated by SD-1 converges slowest. Therefore, we can see that the use of the step sizes 
satisfying the Wolfe-type conditions is a good way to solve fixed point problems by using 
the Krasnosel’skii-Mann algorithm, as seen in Figures [T] and |2| illustrating the behaviors of 
SD-1, SD-2, and SD-3 on Problem 13.II when d := 10^, 10^. Figure [HI indicates the behaviors 
of SD-1, SD-2, and SD-3 when d := 10^. Similarly to what is shown in FigurejSl SD-3 finds 
a fixed point of T faster than SD-1 and SD-2 can. 


29 





























































Number of iterations 



Eiapsed time [sec] 


(a) \\xn — T{xn)\\ VS. no. of itera- (b) \\xn — T(x„)|| vs. elapsed time 
tions 


Figure 7: Evaluation of ||xn — T{xn)\\ in terms of the number of iterations and 
elapsed time for Algorithms FR, PRP+, HS+, DY, and HZ for Problem 13.21 
when d := 10^ 
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(a) \\xn — T{xn)\\ VS. uo. of itera- (b) \\xn — T{xn)\\ vs. elapsed time 
tions 


Figure 8: Evaluation of \\Xn — T{xn)\\ in terms of the number of iterations and 
elapsed time for Algorithms FR, PRP-I-, HS-I-, DY, and HZ for Problem 13.21 
when d := 10^ 


Figure[7l[a) is the evaluation of (Hxn,—T(ai„)||)„gN in terms of the number of iterations for 
Algorithms FR, PRP-j-, HS-I-, DY, and HZ when d := 10^. Except for HS-I-, the algorithms 
approximate the fixed points of T very rapidly. It can be also seen that the algorithms other 
than HS-I- satisfy \\x 2 — F(a; 2 )|| = 0. Figure[71b) is the evaluation of (||a;„ — T(a:„)||)„gN in 
terms of the elapsed time. Here, we can see that FR, PRP-j-, and DY can find fixed points 
of T faster than SD-1 and SD-2 (Figure[5]). Figure |8] indicates the behaviors of FR, PRP-j-, 
HS-I-, DY, and HZ when d := 10^. The results in these figures are almost the same as the 
ones in Figures 0 

From the above numerical results, we can conclude that the proposed algorithms can 
find optimal solutions to Problems l3.1l and l3.2l faster than the previous fixed point algorithms 
can. In particular, it can be seen that the algorithms for which the SRs of Algorithm 13.11 
are high converge quickly to solutions of Problems 13.11 and 13.21 
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4 Conclusion and Future Work 


This paper discussed the fixed point problem for a nonexpansive mapping on a real Hilbert 
space and presented line search fixed point algorithms for solving it on the basis of nonlinear 
conjugate gradient methods for unconstrained optimization and their convergence analyses 
and convergence rate analyses. Moreover, we used these algorithms to solve concrete con¬ 
strained quadratic programming problems and generalized convex feasibility problems and 
numerically compared them with the previous fixed point algorithms based on the Kras- 
nosel’skii-Mann fixed point algorithm. The numerical results showed that the proposed 
algorithms can find optimal solutions to these problems faster than the previous algorithms. 

In the experiment, the line search algorithm 1 Algorithm 13.1|) could not compute appro¬ 
priate step sizes for fixed point algorithms other than Algorithms SD-2, SD-3, and PRP-1-. 
In the future, we should consider modifying the algorithms to enable the line search to 
compute appropriate step sizes. Or we may need to develop new line searches that can be 
applied to all of the fixed point algorithms considered in this paper. 

The main objective of this paper was to devise line-search fixed-point algorithms to 
accelerate the previous Krasnosel’skh-Mann fixed point algorithm defined by i.e., 

Xn+I ■■= XnXn + (1 “ Xn)T{xn) (n S N), where (A„)„gN c [0,1] with - A„) = oo 

and xq G H is an initial point. Another particularly interesting problem is determining 
whether or not there are line search fixed point algorithms to accelerate the following Halpern 
fixed point algorithm [TTI |34] : for all n G N, 

^n+l •— OijiXQ “t“ (1 O^n) T (^n) ; 

where (an)neN C (0,1) satisfies lim„_>oo Q;„ = 0 and Halpern algorithm 

can minimize the convex function || • —xqIP over Fix(T) (see, e.g., [4] Theorem 6.17]). A 
previously reported result [2T1 Theorem 3.1, Proposition 3.2] showed that there is an incon¬ 
venient possibility that the Halpern-type algorithm with a diminishing step size sequence 
(e.g., an := l/(n -I- 1)“, where a G (0,1]) and any of the FR, PRP, HS, and DY formulas 
used in the conventional conjugate gradient methods may not converge to the minimizer 
of II • —a:o|P over Fix(T). However, there is room for further research into devising line 
search fixed point algorithms to accelerate the Halpern algorithm with a diminishing step 
size sequence. 
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